Finding latency through a physical network in a virtualized network

ABSTRACT

Techniques are described for determining latency in a physical network that includes a number of network devices over which packets travel. A virtual network controller receives a plurality of messages from a plurality of network devices in a network, each of the messages including a packet signature comprising a hash of an invariant portion of an original packet that uniquely identifies the original packet, an identifier of one of the plurality of network devices from which the respective message was received, and a timestamp indicating a time an original packet was processed by the network device from which the respective message was received. The virtual network controller determines a latency of a physical network path in the network based on analysis of contents of the identified messages having a common packet signature.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional Application No.61/722,696, filed Nov. 5, 2012; U.S. Provisional Application No.61/721,979, filed Nov. 2, 2012; U.S. Provisional Application No.61/721,994, filed Nov. 2, 2012; U.S. Provisional Application No.61/718,633, filed Oct. 25, 2012; U.S. Provisional Application No.61/656,468, filed Jun. 6, 2012; U.S. Provisional Application No.61/656,469, filed Jun. 6, 2012; and U.S. Provisional Application No.61/656,471, filed Jun. 6, 2012, the entire content of each of whichbeing incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to communication networks.

BACKGROUND

In a typical cloud data center environment, there is a large collectionof interconnected servers that provide computing and/or storage capacityto run various applications. For example, a data center may comprise afacility that hosts applications and services for subscribers, i.e.,customers of data center. The data center may, for example, hosts all ofthe infrastructure equipment, such as networking and storage systems,redundant power supplies, and environmental controls. In a typical datacenter, clusters of storage systems and application servers areinterconnected via high-speed switch fabric provided by one or moretiers of physical network switches and routers. More sophisticated datacenters provide infrastructure spread throughout the world withsubscriber support equipment located in various physical hostingfacilities.

SUMMARY

In general, the disclosure provides techniques for determining latencyin a physical network that includes a number of network devices overwhich packets travel. In a virtual network architecture, informationregarding latency of any particular flow, i.e., the time it takes for apacket to travel from one network device (e.g., server) to anothernetwork device via a particular path of switches and connectors, may notbe readily available to the virtual network.

When a packet matching a defined set of monitored packets travelsthrough a network device (e.g., a switch or router) during the definedtime period, the network device can make a copy of the packet withoutaffecting the flow of the packet, and send information from the copiedpacket back to an analytics engine of a logically centralized virtualnetwork controller along with the time stamp and the identity of thenetwork device. In other words, the analytics engine receivesinformation on when and where the packet has travelled. By analyzingthis information from a number of network devices, analytics engines ofthe virtual network controller can determine the time taken by specificpackets to traverse the physical network, and can identify networkdevices and/or connections in the physical network that slows the speedof the network. Additionally, instead of sending back an entire copy ofthe monitored packet, the network device can take a hash, i.e.,signature, of an invariant portion of the copied packet that uniquelyidentifies the packet, for instance the payload, and send the signatureback to the analytic engine along with a device identifier and timestampinformation. Sending the signatures instead of the entire packet canprovide a more scalable mechanism by compressing the amount ofinformation that needs to be sent and stored in the network.

Using a collection of such latency information, the virtual networkcontroller can identify places in the physical network that are slow orwhere bottlenecks in traffic are occurring. Such a bottleneck may beindicative of a problem with the physical network, such as, for example,a deteriorated cable. Identifying such problems in the physical networkwithout having to run specific testing on each of the components of thenetwork may save time and money, and can help ensure that the networkperforms optimally and without interruption.

In one embodiment, a method for determining latency of a physicalnetwork path in a network includes receiving, by a virtual networkcontroller, a plurality of messages from a plurality of network devicesin a network, wherein each of the messages includes (1) a packetsignature comprising a hash of an invariant portion of an originalpacket that uniquely identifies the original packet, (2) an identifierof one of the plurality of network devices from which the respectivemessage was received, and (3) a timestamp indicating a time an originalpacket was processed by the network device from which the respectivemessage was received. The method also includes identifying, by thevirtual network controller, two or more of the plurality of messageshaving a common packet signature, and determining, by the virtualnetwork controller, a latency of a physical network path in the networkbased on analysis of contents of the identified messages having a commonpacket signature.

In another embodiment, a method includes receiving from a virtualnetwork controller, by a network device, information specifying packetcharacteristics of packets to be analyzed, receiving a packet,responsive to determining that the packet matches the specifiedcharacteristics, and by a virtual network agent executing on the networkdevice, determining a hash of an invariant portion of the packet thatuniquely identifies the packet to obtain a packet signature, andforwarding, to the virtual network controller, a message that specifies:(1) the packet signature, (2) an identifier of the network device, and(3) a timestamp indicating a time the packet was processed by thenetwork device.

In another embodiment, a computer-readable storage medium includesinstructions for causing a programmable processor to receive a pluralityof messages from a plurality of network devices in a network, whereineach of the messages includes (1) a packet signature comprising a hashof an invariant portion of an original packet that uniquely identifiesthe original packet, (2) an identifier of one of the plurality ofnetwork devices from which the respective message was received, and (3)a timestamp indicating a time an original packet was processed by thenetwork device from which the respective message was received, identifytwo or more of the plurality of messages having a common packetsignature, and determine a latency of a physical network path in thenetwork based on analysis of contents of the identified messages havinga common packet signature.

In a further embodiment, a virtual network controller includes one ormore processors, and a plurality of virtual machines executed by theprocessors to receive a plurality of messages from a plurality ofnetwork devices in a network, wherein each of the messages includes (1)a packet signature comprising a hash of an invariant portion of anoriginal packet that uniquely identifies the original packet, (2) anidentifier of one of the plurality of network devices from which therespective message was received, and (3) a timestamp indicating a timean original packet was processed by the network device from which therespective message was received. The virtual network controller alsoincludes a plurality of analytics virtual machines, wherein theplurality of virtual machines identify two or more of the plurality ofmessages having a common packet signature, and determine a latency of aphysical network path in the network based on analysis of contents ofthe identified messages having a common packet signature.

In another example, a system includes a virtual network controller thatincludes one or more processors, a plurality of virtual machinesexecuted by the processors, and a plurality of network devicescomprising one or more processors, wherein the plurality of networkdevices receive from the virtual network controller, informationspecifying packet characteristics of packets to be analyzed receiving apacket, wherein the plurality of virtual machines receive a plurality ofmessages from the plurality of network devices, wherein each of themessages includes (1) a packet signature comprising a hash of aninvariant portion of an original packet that uniquely identifies theoriginal packet, (2) an identifier of one of the plurality of networkdevices from which the respective message was received, and (3) atimestamp indicating a time an original packet was processed by thenetwork device from which the respective message was received, whereinthe virtual network controller further comprises a plurality ofanalytics virtual machines that identify two or more of the plurality ofmessages having a common packet signature, and determine a latency of aphysical network path in the network based on analysis of contents ofthe identified messages having a common packet signature, and whereinthe plurality of network devices comprise a virtual network agentexecuting on the processors that, responsive to determining that thepacket matches the specified characteristics, determining a hash of aninvariant portion of the packet that uniquely identifies the packet toobtain a packet signature, and forward, to the virtual networkcontroller, a message that specifies: (1) the packet signature, (2) anidentifier of the network device, and (3) a timestamp indicating a timethe packet was processed by the network device.

The details of one or more aspects of this disclosure are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description anddrawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example data center in whichexamples of the techniques described herein may be implemented.

FIG. 2 is a block diagram illustrating in further detail an examplesystem in which the techniques described herein may be implemented.

FIG. 3 is another block diagram illustrating an example system 50illustrating example configuration of chassis switch and TOR switches asdescribed herein.

FIG. 4 is a block diagram illustrating an example implementation of avirtual network controller for facilitating operation of one or morevirtual networks in accordance with one or more embodiments of thisdisclosure.

FIG. 5 is a block diagram illustrating an example implementation of avirtual network controller for facilitating operation of one or morevirtual networks in accordance with one or more embodiments of thisdisclosure.

FIGS. 6-7 are flowcharts illustrating example operations of networkdevices in accordance with one or more embodiments of this disclosure.

FIG. 8 is a block diagram illustrating an example device in accordancewith one or more aspects of the present disclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example network 8 having adata center 10 in which examples of the techniques described herein maybe implemented. In general, data center 10 provides an operatingenvironment for applications and services for customers 11 coupled tothe data center by service provider network 12. Data center 5 may, forexample, host infrastructure equipment, such as networking and storagesystems, redundant power supplies, and environmental controls. Serviceprovider network 12 may be coupled to one or more networks administeredby other providers, and may thus form part of a large-scale publicnetwork infrastructure, e.g., the Internet.

In some examples, data center 10 may represent one of manygeographically distributed network data centers. As illustrated in theexample of FIG. 1, data center 10 may be a facility that providesnetwork services for customers 11. Customers 11 may be collectiveentities such as enterprises and governments or individuals. Forexample, a network data center may host web services for severalenterprises and end users. Other exemplary services may include datastorage, virtual private networks, traffic engineering, file service,data mining, scientific- or super-computing, and so on. In someembodiments, data center 10 may be individual network servers, networkpeers, or otherwise.

In this example, data center 5 includes set of storage systems andapplication servers 12A-12X (herein, “servers 12”) interconnected viahigh-speed switch fabric 14 provided by one or more tiers of physicalnetwork switches and routers. Switch fabric 14 is provided by a set ofinterconnected top-of-rack (TOR) switches 16A-16BN (“TOR switches” 16)coupled to a distribution layer of chassis switches 18. Although notshown, data center 10 may also include, for example, one or morenon-edge switches, routers, hubs, gateways, security devices such asfirewalls, intrusion detection, and/or intrusion prevention devices,servers, computer terminals, laptops, printers, databases, wirelessmobile devices such as cellular phones or personal digital assistants,wireless access points, bridges, cable modems, application accelerators,or other network devices.

In this example, TOR switches 16 and chassis switches 18 provide servers12 with redundant (multi-homed) connectivity to IP fabric 20 and serviceprovider network 12. Chassis switches 18 aggregates traffic flows andprovides high-speed connectivity between TOR switches 16. TOR switches16A and 16B may be network devices that provide layer 2 (MAC address)and/or layer 3 (IP address) routing and/or switching functionality. TORswitches 16 and chassis switches 18 may each include one or moreprocessors and a memory, and that are capable of executing one or moresoftware processes. Chassis switches 18 are coupled to IP fabric 20,which performs layer 3 routing to route network traffic between datacenter 10 and customers 11 using service provider network 12.

Virtual network controller 22 (“VNC”) provides a logically centralizedcontroller for facilitating operation of one or more virtual networkswithin data center 10 in accordance with one or more embodiments of thisdisclosure. In some examples, virtual network controller 22 may operatein response to configuration input received from network administrator24. As described in further detail below, servers 12 may include one ormore virtual switches that create and manage one or more virtualnetworks that are used by applications.

Typically, the traffic between any two network devices, such as betweennetwork devices within IP fabric 20 (not shown) or between servers 12and customers 11, for example, can traverse the physical network usingmany different paths. For example, there may be several different pathsof equal cost between two network devices. In some cases, packetsbelonging to network traffic from one network device to the other may bedistributed among the various possible paths using a routing strategycalled multi-path routing at each network switch node. For example, theInternet Engineering Task Force (IETF) RFC 2992, “Analysis of anEqual-Cost Multi-Path Algorithm,” describes a routing technique forrouting packets along multiple paths of equal cost. The techniques ofRFC 2992 analyzes one particular multipath routing strategy involvingthe assignment of flows to bins by hashing packet header fields thatsends all packets from a particular network flow over a singledeterministic path.

For example, a “flow” can be defined by the five values used in a headerto a packet, or “five-tuple,” i.e., the protocol, Source IP address,Destination IP address, Source port and Destination port that are usedto route packets through the physical network. For example, the protocolspecifies the communications protocol, such as TCP or UDP, and Sourceport and Destination port refer to source and destination ports of theconnection. A set of one or more packet data units (PDUs) that match aparticular flow entry represent a flow. Flows may be broadly classifiedusing any parameter of a PDU, such as source and destination MAC and IPaddresses, a Virtual Local Area Network (VLAN) tag, transport layerinformation, a Multiprotocol Label Switching (MPLS) or Generalized MPLS(GMPLS) label, and an ingress port of a network device receiving theflow. For example, a flow may be all PDUs transmitted in a TransmissionControl Protocol (TCP) connection, all PDUs sourced by a particular MACaddress or IP address, all PDUs having the same VLAN tag, or all PDUsreceived at the same switch port.

Each individual switch router in the network will perform its ownindependent hashing computation to determine the path that will be usedby a particular flow. The ECMP paths between the first and secondnetwork devices may be viewed by the virtual network as one physicalconnection, as their packet (inner packet) is encapsulated by the outerIP encapsulation.

In such a network, information regarding network controller 22. To findthe real latency, a statistical technique may need to be employed.Typically, the virtual network domain is controlled separately from thephysical network domain and, as a result, the ability to ascertain theactual path of a packet for a given network flow within the virtualnetwork domain is not straightforward, and typically requires knowledgeof the state of the physical network.

One technique that could be employed to determine the path taken by anetwork IP packet through a switch router network is to use an IPtrace-route function, which is supported by most operating systems aswell as network operating systems. However, such a trace-route functiondoes not work well when a multi-path latency of any particular flow,i.e., the time it takes for a packet to travel from one network device(e.g., server) to another network device via a particular path ofswitches and connectors (e.g., within IP fabric 20), is not readilyavailable to the virtual network and to virtual routing technique isemployed, as different network flows use different paths through thenetwork and the trace-route packet that is used to ascertain the routewill not have the same header as the application packet. Because hashingfunctions in most network switch routers depends on the packet header,this trace-route packet may not follow the same path.

In accordance with the techniques of this disclosure, one method fordetermining latency in a multi-path routing environment is to collectstatistics on every virtual switch node (e.g., residing on servers 12),that is, every switch node collects data on which packets have travelledthrough the switch node and when they travelled through the switch nodebetween servers 12. The switch then sends this data to an analyticsengine executing on virtual network controller 22. The analytics enginecan use the data to calculate latency. Collecting all such data from thevirtual switch nodes on all servers 12, however, may result in massiveamounts of data, which may be difficult to use effectively and will eataway network bandwidth. So in some exemplary embodiments, anadministrator 24 may choose to restrict the data that is gathered. Forexample, the administrator 24 may specify at virtual network controller22 that statistics are to be captured for a certain class of traffic,and may also restrict the period of time over which statistic arecollected. To capture a certain class of traffic the packet can be usedlike a match list, called a packet classifier. Virtual networkcontroller 22 can send the packet classifiers down to be installed onthe appropriate servers 12.

FIG. 2 is a block diagram illustrating an example implementation of datacenter 10 of FIG. 1 in further detail. In the example of FIG. 2, datacenter 10 includes an overlay network that extends switch fabric 14 fromphysical switches 16, 18 to software switches 30A-30X (also referred toas a “virtual switches). Virtual switches 30 dynamically create andmanage one or more virtual networks 34 to be used by applicationscommunicating with application instances. In one example, virtualswitches 30 execute the virtual network as an overlay network, whichprovides the capability to decouple an application's virtual addressfrom a physical address (e.g., IP address) of the one of servers 12A-12X(“servers 12”) on which the application is executing. Each virtualnetwork 34 may use its own addressing and security scheme and may beviewed as orthogonal from the physical network and its addressingscheme. Various techniques may be used to transport packets within andacross virtual network(s) 34 over the physical network.

Each virtual switch 30 may execute within a hypervisor, a host operatingsystem or other component of each of servers 12. In the example of FIG.2, virtual switch 30 executes within hypervisor 31, also often referredto as a virtual machine manager (VMM), which provides a virtualizationplatform that allows multiple operating systems to concurrently run onone of host servers 12. In the example of FIG. 2, virtual switch 30Amanages virtual networks 34, each of which provides a networkenvironment for execution of one or more virtual machines (VMs) 36 ontop of the virtualization platform provided by hypervisor 31. Each VM 36is associated with one of the virtual subnets VN0-VN2 managed by thehypervisor 31.

In general, each VM 36 may be any type of software application and maybe assigned a virtual address for use within a corresponding virtualnetwork 34, where each of the virtual networks may be a differentvirtual subnet provided by virtual switch 30A. A VM 36 may be assignedits own virtual layer three (L3) IP address, for example, for sendingand receiving communications but may be unaware of an IP address of thephysical server 12A on which the virtual machine is executing. In thisway, a “virtual address” is an address for an application that differsfrom the logical address for the underlying, physical computer system,i.e., server 12A in the example of FIG. 2.

In one implementation, each of servers 12 includes a virtual networkagent (“VN agent”) 35A-35X (“VN agents 35”) that controls the overlay ofvirtual networks 34 and that coordinates the routing of data packetswithin server 12. In general, each VN agent 35 communicates with virtualnetwork controller 22, which generates commands to control routing ofpackets through data center 10. VN agents 35 may operate as a proxy forcontrol plane messages between virtual machines 36 and virtual networkcontroller 22. For example, a VM 36 may request to send a message usingits virtual address via the VN agent 35A, and VN agent 35A may in turnsend the message and request that a response to the message be receivedfor the virtual address of the VM 36 that originated the first message.In some cases, a VM 36 may invoke a procedure or function call presentedby an application programming interface of VN agent 35A, and the VNagent 35A may handle encapsulation of the message as well, includingaddressing.

In one example, network packets, e.g., layer three (L3) IP packets orlayer two (L2) Ethernet packets generated or consumed by the instancesof applications executed by virtual machines 36 within the virtualnetwork domain may be encapsulated in another packet (e.g., another IPor Ethernet packet) that is transported by the physical network. Thepacket transported in a virtual network may be referred to herein as an“inner packet” while the physical network packet may be referred toherein as an “outer packet.” Encapsulation and/or de-capsulation ofvirtual network packets within physical network packets may be performedwithin virtual switches 30, e.g., within the hypervisor or the hostoperating system running on each of servers 12. As another example,encapsulation and de-capsulation functions may be performed at the edgeof switch fabric 14 at a first-hop TOR switch 16 that is one hop removedfrom the application instance that originated the packet. Thisfunctionality is referred to herein as tunneling and may be used withindata center to create one or more overlay networks. Other exampletunneling protocols may be used, including IP over GRE, VxLAN, MPLS overGRE, etc.

As noted above, virtual network controller 22 provides a logicallycentralized controller for facilitating operation of one or more virtualnetworks within data center 10. Virtual network controller 22 may, forexample, maintain a routing information base, e.g., on or more routingtables that store routing information for the physical network as wellas the overlay network of data center 10. Similarly, switches 16, 18 andvirtual switches 30 maintain routing information, such as one or morerouting and/or forwarding tables. In one example implementation, virtualswitch 30A of hypervisor 31 implements a network forwarding table (NFT)32 for each virtual network 34. In general, each NFT 32 storesforwarding information for the corresponding virtual network 34 andidentifies where data packets are to be forwarded and whether thepackets are to be encapsulated in a tunneling protocol, such as with oneor more outer IP addresses.

The routing information may, for example, map packet key information(e.g., destination IP information and other select information frompacket headers) to one or more specific next hops within the networksprovided by virtual switches 30 and switch fabric 14. In some case, thenext hops may be chained next hop that specify a set of operations to beperformed on each packet when forwarding the packet, such as may be usedfor flooding next hops and multicasting replication. In some cases,virtual network controller 22 maintains the routing information in theform of a radix tree having leaf nodes that represent destinationswithin the network. U.S. Pat. No. 7,184,437 provides details on anexemplary embodiment of a router that utilizes a radix tree for routeresolution, the contents of U.S. Pat. No. 7,184,437 being incorporatedherein by reference in its entirety.

As shown in FIG. 2, each virtual network 34 provides a communicationframework for encapsulated packet communications 37 for the overlaynetwork established through switch fabric 14. In this way, networkpackets associated with any of virtual machines 36 may be transported asencapsulated packet communications 37 via the overlay network. Inaddition, in the example of FIG. 2, each virtual switch 30 includes adefault network forwarding table NFT₀ and provides a default route thatallows packet to be forwarded to virtual subnet VN0 withoutencapsulation, i.e., non-encapsulated packet communications 39 per therouting rules of the physical network of data center 10. In this way,subnet VN0 and virtual default network forwarding table NFT₀ provide amechanism for bypassing the overlay network and sending non-encapsulatedpacket communications 39 to switch fabric 14.

Moreover, virtual network controller 22 and virtual switches 30 maycommunicate using virtual subnet VN0 in accordance with default networkforwarding table NFT₀ during discovery and initialization of the overlaynetwork, and during conditions where a failed link has temporarilyhalted communication via the overlay network. In some aspects, onceconnectivity with the virtual network controller 22 is established, thevirtual network controller 22 updates its local routing table to takeinto account new information about any failed links and directs virtualswitches 30 to update their local network forwarding tables 32. Forexample, virtual network controller 22 may output commands to virtualnetwork agents 35 to update one or more NFTs 32 to direct virtualswitches 30 to change the tunneling encapsulation so as to re-routecommunications within the overlay network, for example to avoid a failedlink.

When link failure is detected, a virtual network agent 35 local to thefailed link (e.g., VN Agent 35A) may immediately change theencapsulation of network packet to redirect traffic within the overlaynetwork and notifies virtual network controller 22 of the routingchange. In turn, virtual network controller 22 updates its routinginformation any may issues messages to other virtual network agents 35to update local routing information stored by the virtual network agentswithin network forwarding tables 32.

In accordance with the techniques of this disclosure, administrator 24may configure packet classifiers to specify which packets are to bemonitored for latency and on which time domains on virtual networkcontroller 22 via commands entered in web console 42. Virtual networkcontroller 22 notifies relevant VN agents 35 of the packet monitoringdefinitions based on the packet classifiers. VN agents 35 install packetcapture logic on respective virtual switches 30. Virtual switches 30match packets using the packet capture logic, and sends copies of thematching packets to VN agents 35. VN agents 35 calculate a packetsignature for each packet, and send information to virtual networkcontroller 22, such as information specifying the packet signature, aswitch identifier of the virtual switch 30 that matched the packets, anda timestamp indicating the time of calculating the packet signature (ora time of matching the packets, for example). Distributed analyticsengines of virtual network controller 22 analyze the receivedinformation and compile results regarding packet latency, as describedin further detail below. Virtual network controller 22 may send results,such as a report, to web console 42 for display to administrator 24.

FIG. 3 is another block diagram illustrating an example system 50illustrating example configuration of routing information within chassisswitch and TOR switches as described herein. System 50 of FIG. 3 may,for example, correspond to portions of data center 10 illustrated inFIGS. 1 and 2.

In this example, chassis switch 52 (“CH 52”), which may be any ofchassis switches 18 of FIG. 1, is coupled to Top of Rack (TOR) switches58A-58B (“TORs 58”) by chassis link 60A and chassis link 60B,respectively (“chassis links 60”). TORs 58 may, in some examples, be anyof TORs 16 of FIG. 1. In the example of FIG. 3, TORs 58 are also coupledto servers 50A-50B (“servers 50”) by TOR links 62A-62D (“TOR links 62”).Servers 50 may be any of servers 210 (FIG. 1). Here, servers 50communicate with both TORs 58, and can physically reside in eitherassociated rack. TORs 58 each communicate with a number of networkswitches, including chassis switch 18A.

Chassis switch 18A has a processor 54A in communication with aninterface for communication with a network as shown, as well as a busthat connects a memory (not shown) to processor 54A. The memory maystore a number of software modules. These modules include software thatcontrols network routing, such as an Open Shortest Path First (OSPF)module (not shown) containing instructions for operating the chassisswitch 18A in compliance with the OSPF protocol. Chassis switch 18Amaintains routing table (“RT table”) 56A containing routing informationfor packets, which describes a topology of a network. Routing table 56Amay be, for example, a table of packet destination Internet protocol(IP) addresses and the corresponding next hop, e.g., expressed as a linkto a network component.

TORs 58 each have a respective processor 54B, 54C, an interface incommunication with chassis switch 18A, and a memory (not shown). Eachmemory contains software modules including an OSPF module and routingtable 56B, 56C as described above.

TORs 58 and chassis switch 18A may exchange routing informationspecifying available routes, such as by using a link-state routingprotocol such as OSPF or IS-IS. TORs 58 may be configured as owners ofdifferent routing subnets. For example, TOR 58A is configured as theowner of Subnet 1, which is the subnet 10.10.10.0/24 in the example ofFIG. 2, and TOR 58A is configured as the owner of Subnet 2, which is thesubnet 10.10.11.0/24 in the example of FIG. 2. As owners of theirrespective Subnets, TORs 58 locally store the individual routes fortheir subnets and need not broadcast all route advertisements up tochassis switch 18A. Instead, in general TORs 58 will only advertisetheir subnet addresses to chassis switch 18A.

Chassis switch 18A maintains a routing table (“RT table”) 56A, whichincludes routes expressed as subnets reachable by TORs 58, based onroute advertisements received from TORs 58. In the example of FIG. 2, RTtable 56A stores routes indicating that traffic destined for addresseswithin the subnet 10.10.11.0/24 can be forwarded on link 60B to TOR 58B,and traffic destined for addresses within the subnet 10.10.10.0/24 canbe forwarded on link 60A to TOR 58A.

In typical operation, chassis switch 18A receives Internet Protocol (IP)packets through its network interface, reads the packets' destination IPaddress, looks up these addresses on routing table 56A to determine thecorresponding destination component, and forwards the packetsaccordingly. For example, if the destination IP address of a receivedpacket is 10.10.0.0, i.e., the address of the subnet of TOR 58A, therouting table of chassis switch 18A indicates that the packet is to besent to TOR 58A via link 60A, and chassis switch 18A transmits thepacket accordingly, ultimately for forwarding to a specific one of theservers 50.

Similarly, each of TORs 58 receives Internet Protocol (IP) packetsthrough its network interface, reads the packets' destination IPaddress, looks up these addresses on its routing table 56 to determinethe corresponding destination component, and forwards the packetsaccording to the result of the lookup.

FIG. 4 is a block diagram illustrating an example implementation of avirtual network controller 22 for facilitating operation of one or morevirtual networks in accordance with one or more embodiments of thisdisclosure. Virtual network controller 22 may, for example, correspondto virtual network controller 22 of data center 10 of FIGS. 1 and 2.

Virtual network controller (VNC) 22 of FIG. 4 illustrates a distributedimplementation of a VNC that includes multiple VNC nodes 80A-80N(collectively, “VNC nodes 80”) to execute the functionality of a datacenter VNC, including managing the operation of virtual switches for oneor more virtual networks implemented within the data center. Each of VNCnodes 80 may represent a different server of the data center, e.g., anyof servers 12 of FIGS. 1-2, or alternatively, on a server or controllercoupled to the IP fabric by, e.g., an edge router of a service providernetwork or a customer edge device of the data center network. In someinstances, some of VNC nodes 80 may execute as separate virtual machineson the same server.

Each of VNC nodes 80 may control a different, non-overlapping set ofdata center elements, such as servers, individual virtual switchesexecuting within servers, individual interfaces associated with virtualswitches, chassis switches, TOR switches, and/or communication links.VNC nodes 80 peer with one another using peering links 86 to exchangeinformation for distributed databases, including distributed databases82A-82K (collectively, “distributed databases 82”), and routinginformation (e.g., routes) for routing information bases 84A-84N(collectively, “RIBs 84”). Peering links 86 may represent peering linksfor a routing protocol, such as a Border Gateway Protocol (BGP)implementation, or another peering protocol by which VNC nodes 80 maycoordinate to share information according to a peering relationship.

VNC nodes 80 of VNC 22 include respective RIBs 84 each having, e.g., oneor more routing tables that store routing information for the physicalnetwork and/or one or more overlay networks of the data centercontrolled by VNC 22. In some instances, one of RIBs 84, e.g., RIB 84A,may store the complete routing table for any of the virtual networksoperating within the data center and controlled by the corresponding VNCnode 80 (e.g., VNC node 80A).

In general, distributed databases 82 define the configuration ordescribe the operation of virtual networks by the data center controlledby distributed VNC 22. For instance, distributes databases 82 mayinclude databases that describe a configuration of one or more virtualnetworks, the hardware/software configurations and capabilities of datacenter servers, performance or diagnostic information for one or morevirtual networks and/or the underlying physical network, the topology ofthe underlying physical network including server/chassis switch/TORswitch interfaces and interconnecting links, and so on. Distributeddatabases 82 may each be implemented using, e.g., a distributed hashtable (DHT) to provide a lookup service for key/value pairs of thedistributed database stored by different VNC nodes 22.

In accordance with the techniques of this disclosure, when virtualnetwork controller 22 notifies VN agents 35 of the servers 12 of thepacket classifier information, and when VN agents 35 send packetsignatures back up to virtual network controller 22, thesecommunications may occur over peering links 66, such as via a routingprotocol like BGP or other peering protocol. Analytics engines ofvirtual network controller 22 may analyze the signature data based ondistributed databases 82, as described in further detail below.

FIG. 5 is a block diagram illustrating an example implementation of avirtual network controller 100 for facilitating operation of one or morevirtual networks in accordance with one or more embodiments of thisdisclosure. Virtual network controller 100 may, for example, correspondto virtual network controller 22 of data center 10 of FIGS. 1 and 2 orvirtual network controller 22 of FIG. 4.

As illustrated in the example of FIG. 5, distributed virtual networkcontroller (VNC) 100 includes one or more virtual network controller(“VNC”) nodes 102A-102N (collectively, “VNC nodes 102”). Each of VNCnodes 102 may represent any of VNC nodes 80 of virtual networkcontroller 22 of FIG. 4. VNC nodes 102 that peer with one anotheraccording to a peering protocol operating over network 160. Network 160may represent an example instance of switch fabric 14 and/or IP fabric20 of FIG. 1. In the illustrated example, VNC nodes 102 peer with oneanother using a Border Gateway Protocol (BGP) implementation, an exampleof a peering protocol. VNC nodes 102 provide, to one another using thepeering protocol, information related to respective elements of thevirtual network managed, at least in part, by the VNC nodes 102. Forexample, VNC node 102A may manage a first set of one or more serversoperating as virtual network switches for the virtual network. VNC node102A may send information relating to the management or operation of thefirst set of servers to VNC node 102N by BGP 118A. For example,referring to FIG. 2, when virtual network controller 22 notifies VNagents 35 of the servers 12 of the packet classifier information, andwhen VN agents 35 send packet signatures back up to virtual networkcontroller 22, these communications may occur as interactions betweenVNC nodes 102 by BGP 118A, for example.

Other elements managed by VNC nodes 102 may include network controllersand/or appliances, network infrastructure devices (e.g., L2 or L3switches), communication links, firewalls, and VNC nodes 102, forexample. Because VNC nodes 102 have a peer relationship, rather than amaster-slave relationship, information may be sufficiently easily sharedbetween the VNC nodes 102. In addition, hardware and/or software of VNCnodes 102 may be sufficiently easily replaced, providing satisfactoryresource fungibility.

Each of VNC nodes 102 may include substantially similar components forperforming substantially similar functionality, said functionality beingdescribed hereinafter primarily with respect to VNC node 102A. VNC node102A may include an analytics database 106A for storing diagnosticinformation related to a first set of elements managed by VNC node 102A.VNC node 102A may share at least some diagnostic information related toone or more of the first set of elements managed by VNC node 102A andstored in analytics database 106, as well as to receive at least somediagnostic information related to any of the elements managed by othersof VNC nodes 102. Analytics database 106A may represent a distributedhash table (DHT), for instance, or any suitable data structure forstoring diagnostic information for network elements in a distributedmanner in cooperation with others of VNC nodes 102. Analytics databases106A-106N (collectively, “analytics databases 106”) may represent, atleast in part, one of distributed databases 82 of distributed virtualnetwork controller 22 of FIG. 4.

VNC node 102A may include a configuration database 110A for storingconfiguration information related to a first set of elements managed byVNC node 102A. Control plane components of VNC node 102A may storeconfiguration information to configuration database 110A using interface144A, which may represent an Interface for Metadata Access Points(IF-MAP) protocol implementation. VNC node 102A may share at least someconfiguration information related to one or more of the first set ofelements managed by VNC node 102A and stored in configuration database110A, as well as to receive at least some configuration informationrelated to any of the elements managed by others of VNC nodes 102.Configuration database 110A may represent a distributed hash table(DHT), for instance, or any suitable data structure for storingconfiguration information for network elements in a distributed mannerin cooperation with others of VNC nodes 102. Configuration databases110A-110N (collectively, “configuration databases 110”) may represent,at least in part, one of distributed databases 82 of distributed virtualnetwork controller 22 of FIG. 4.

Virtual network controller 100 may perform any one or more of theillustrated virtual network controller operations represented by modules130, which may include orchestration 132, user interface 134, VNC globalload balancing 136, and one or more applications 138. VNC 100 executesorchestration module 132 to facilitate the operation of one or morevirtual networks in response to a dynamic demand environment by, e.g.,spawning/removing virtual machines in data center servers, adjustingcomputing capabilities, allocating network storage resources, andmodifying a virtual topology connecting virtual switches of a virtualnetwork. VNC global load balancing 136 executed by VNC 100 supports loadbalancing of analytics, configuration, communication tasks, e.g., amongVNC nodes 102. Applications 138 may represent one or more networkapplications executed by VNC nodes 102 to, e.g., change topology ofphysical and/or virtual networks, add services, or affect packetforwarding.

User interface 134 includes an interface usable to an administrator (orsoftware agent) to control the operation of VNC nodes 102. For instance,user interface 134 may include methods by which an administrator maymodify, e.g. configuration database 110A of VNC node 102A.Administration of the one or more virtual networks operated by VNC 100may proceed by uniform user interface 134 that provides a single pointof administration, which may reduce an administration cost of the one ormore virtual networks.

VNC node 102A may include a control plane virtual machine (VM) 112A thatexecutes control plane protocols to facilitate the distributed VNCtechniques described herein. Control plane VM 112A may in some instancesrepresent a native process. In the illustrated example, control VM 112Aexecutes BGP 118A to provide information related to the first set ofelements managed by VNC node 102A to, e.g., control plane virtualmachine 112N of VNC node 102N. Control plane VM 112A may use an openstandards based protocol (e.g., BGP based L3VPN) to distributeinformation about its virtual network(s) with other control planeinstances and/or other third party networking equipment(s). Given thepeering based model according to one or more aspects described herein,different control plane instances (e.g., different instances of controlplane VMs 112A-112N) may execute different software versions. In one ormore aspects, e.g., control plane VM 112A may include a type of softwareof a particular version, and the control plane VM 112N may include adifferent version of the same type of software. The peeringconfiguration of the control node devices may enable use of differentsoftware versions for the control plane VMs 112A-112N. The execution ofmultiple control plane VMs by respective VNC nodes 102 may prevent theemergence of a single point of failure.

Control plane VM 112A communicates with virtual network switches, e.g.,illustrated VM switch 174 executed by server 140, using a communicationprotocol operating over network 160. Virtual network switches facilitateoverlay networks in the one or more virtual networks. In the illustratedexample, control plane VM 112A uses Extensible Messaging and PresenceProtocol (XMPP) 116A to communicate with at least virtual network switch174 by XMPP interface 150A. Virtual network route data, statisticscollection, logs, and configuration information may in accordance withXMPP 116A be sent as XML documents for communication between controlplane VM 112A and the virtual network switches. Control plane VM 112Amay in turn route data to other XMPP servers (such as an analyticscollector) or may retrieve configuration information on behalf of one ormore virtual network switches. Control plane VM 112A may further executea communication interface 144A for communicating with configurationvirtual machine (VM) 108A associated with configuration database 110A.Communication interface 144A may represent an IF-MAP interface.

VNC node 102A may further include configuration VM 108A to storeconfiguration information for the first set of element to and manageconfiguration database 110A. Configuration VM 108A, although describedas a virtual machine, may in some aspects represent a native processexecuting on an operating system of VNC node 102A. Configuration VM 108Aand control plane VM 112A may communicate using IF-MAP by communicationinterface 144A and using XMPP by communication interface 146A. In someaspects, configuration VM 108A may include a horizontally scalablemulti-tenant IF-MAP server and a distributed hash table (DHT)-basedIF-MAP database that represents configuration database 110A. In someaspects, configuration VM 108A may include a configuration translator,which may translate a user friendly higher-level virtual networkconfiguration to a standards based protocol configuration (e.g., a BGPL3VPN configuration), which may be stored using configuration database110A. Communication interface 140 may include an IF-MAP interface forcommunicating with other network elements. The use of the IF-MAP maymake the storage and management of virtual network configurations veryflexible and extensible given that the IF-MAP schema can be dynamicallyupdated. Advantageously, aspects of virtual network controller 100 maybe flexible for new applications 138.

VNC node 102A may further include an analytics virtual machine (VM) 104Ato store diagnostic information (and/or visibility information) relatedto at least the first set of elements managed by VNC node 102A. Controlplane VM and analytics VM 104 may communicate using an XMPPimplementation by communication interface 146A. Analytics VM 104A,although described as a virtual machine, may in some aspects represent anative process executing on an operating system of VNC node 102A.

Analytics VM 104A may include analytics database 106A, which mayrepresent an instance of a distributed database that stores visibilitydata for virtual networks, such as one of distributed database 82 ofdistributed virtual network controller 22 of FIG. 4. Visibilityinformation may describe visibility of both distributed VNC 100 itselfand of customer networks. The distributed database may include an XMPPinterface on a first side and a REST/JASON/XMPP interface on a secondside.

Virtual network switch 174 may implement the layer 3 forwarding andpolicy enforcement point for one or more end points and/or one or morehosts. The one or more end points or one and/or one or more hosts may beclassified into a virtual network due to configuration from controlplane VM 112A. Control plane VM 112A may also distributevirtual-to-physical mapping for each end point to all other end pointsas routes. These routes may give the next hop mapping virtual IP tophysical IP and encapsulation technique used (e.g., one of IPinIP,NVGRE, VXLAN, etc.). Virtual network switch 174 may be agnostic toactual tunneling encapsulation used. Virtual network switch 174 may alsotrap interesting layer 2 (L2) packets, broadcast packets, and/orimplement proxy for the packets, e.g. using one of Address ResolutionProtocol (ARP), Dynamic Host Configuration Protocol (DHCP), Domain NameService (DNS), etc.

In some cases, different VNC nodes 102 may be provided by differentsuppliers. However, the peering configuration of VNC nodes 102 mayenable use of different hardware and/or software provided by differentsuppliers for implementing the VNC nodes 102 of distributed VNC 100. Asystem operating according to the techniques described above may providelogical view of network topology to end-host irrespective of physicalnetwork topology, access type, and/or location. Distributed VNC 100provides programmatic ways for network operators and/or applications tochange topology, to affect packet forwarding, and/or to add services, aswell as horizontal scaling of network services, e.g. firewall, withoutchanging the end-host view of the network.

In accordance with the techniques of this disclosure, analytics VM 104(which may also be referred to herein as “analytics engines”) analyzethe status of the physical network indicated by network 160, which mayinclude IP fabric 20 (FIG. 1). Network 160 may include, for example,switches (routers) and connectors. Analytics VM 104 includes, forexample, analytics databases 106 and memory (not shown) that may belinearly scaled via same virtual network forming the network 160.Analytics VM 104 are connected to network 160 via connectors 148A-148N.The system of FIG. 5 includes various servers including server 170,which may be servers such as described in FIG. 2 as servers 12. Server170 includes a virtual switch 174, also sometimes referred to as avirtual network router (VN-router), which encapsulates and forwards theapplication packets over the physical network, and a virtual networkagent, VN switch agent 172, which provides the intelligence to virtualswitch 174 by talking to virtual network controller 100 and providesstatistics to analytics VM 104. The virtual switch 174 hides thephysical network from the physical switches and routers as found in theIP fabric of network 160. Thus, it can appear that, for example, server12A is directly connected to server 12N (FIG. 1). Servers 12 alsoinclude a series of guest virtual machines VM 36 (FIG. 2). In someexamples, analytics VM 104 are actually some instances of VMs 36.

One method for determining latency in a multi-path routing environmentis to collect statistics on every virtual switch 174, that is, everyvirtual switch 174, collects data on which packets have travelledthrough the virtual switch 174 and when they travelled through thevirtual switch 174 between servers 170. The virtual switch 174 thensends this data to one of the analytics VM 104. The analytics VMs 104can use the data to calculate latency. For example, the administratormay specify that statistics are to be captured for a certain class oftraffic, and may also restrict the period of time over which statisticare collected. To capture a certain class of traffic the packet can beused like a match list, called a packet classifier.

For example, to see how certain flows are doing, an example packetclassifier “PC” is defined as:

  PC = {    Id: 4502  Start-time: 8:00:00 12/5/2012  End-time: 8:01:0012/5/2012  SEQ: {   Protocol: ‘TCP’,   SIP: 10.1.1.0/24,   SP: any,  DIP: 10.5.1.42/32,   DP: 80,   Comment: ‘all web traffic to loadbalancer’  },  {   Protocol: ‘TCP’ ,   SIP: 10.1.1.42/32,   SP: any,  DIP: 10.5.1.45/32,   DP: any,   Comment: ‘all traffic from loadbalancer to firewall’  }  }

This will capture web traffic to load balancer and traffic from the loadbalancer that are sent to firewall, starting on 8 am Dec. 5, 2012 forone minute. This classifier can be set by web console 42 (FIG. 2) toanalytics VM 104 which will inform all relevant VN switch agents 172.

FIG. 6 is a flowchart illustrating example operation of network devicesin accordance with the techniques of this disclosure. FIG. 6 isdescribed with respect to FIGS. 1-3. As shown in FIG. 6, to determinethe latency of a class of traffic, a request is initiated by the admin,such as via a device used to access the network, for instance, webconsole 42. The request may specify a packet classifier that describescharacteristics of packets to be analyzed, and may also define a timeperiod in which the packet classifier should be applied, such as bysetting a start time and end time. The web console 42 delivers a messageto the analytics VM 104 with the packet classifier information (500).

The Analytics VM 104, in turn, receives the message (502) and notifiesand delivers the packet classifier and time period securely to theappropriate VN switch agent 172 in the network (504). Analytics VM 104can identify which VN switch agents 172 need to be notified based on thepacket classifier, such as based on a comparison of the IP addresses inthe packet classifier relative to which subnets are owned by the VNswitch agent 172. Each of the notified VN switch agents 172 can installthis packet classifier on their respective virtual switch 174 to capturethe appropriate packets, e.g., at their egress interface. Each virtualswitch 174 thus can enable the received packet classifier at thespecified start time. If the end time is in the past, virtual switch 174can ignore this packet classifier. If start time is in the past, virtualswitch 174 can enable the packet classifier immediately. The virtualswitch 174 will disable the packet classifier at the end time.

When a packet traveling in the network matches a packet classifier onthe virtual switch 174 (509), the virtual switch 174 sends a copy of thepacket to a slow path for processing at the VN switch agent 172 (510),without affecting delivery of the original received packet. In anetworking data path of the switches and router, when a packet comes tobe forwarded, there may exist two paths, fast path and slow path. Fastpath is like cached memory, and determines what to do with the packet,such as where to send it to, without delay. If the information is not athand, for example like cache miss, the packet is queued for furtherprocessing, where some other program looks up database to what to dowith this packet, and if necessary, update the fast path cache.

Usually a network device performs this flow-based forwarding by cachingor otherwise storing flow state for the packet flows of a givencommunication session between two devices. Generally, upon recognizing afirst packet of a new flow, a network device initializes data to recordthe state data for the session. The VN switch agent 172 may inspectpacket flows for the sessions. In some cases, the VN switch agent 172may comprise two forwarding paths, a first path for processing a firstpacket of a newly established flow and a second path for inspecting andforwarding subsequent packets associated with a pre-existing flow. Thefirst path through the VN switch agent 172 may be referred to as the“first path,” “slow path,” or “session management path.” At this time,after processing the first packet of the newly established flow, the VNswitch agent 172 may update flow tables to record the session andotherwise initialize session data. The second path through VN switchagent 172 may be referred to as the “fast path” because the second pathnormally does not take as long to traverse as the first path due to thelack of detailed inspection needed for subsequent packets in analready-established flow. Further details relating to network deviceshaving a fast path and slow path can be found in U.S. Pat. No.8,339,959, filed Jul. 30, 2008, entitled “Streamlined Packet Forwardingusing Dynamic Filters for Routing and Security in a Shared ForwardingPlane,” the entire content of which is incorporated by reference herein.Virtual switch 174 sends additional information such as a timestamp,ingress port and egress port etc. to the slow path along with the copyof the packet.

As will be described in more detail below with respect to FIG. 7 the VNswitch agent 172 calculates the packet signature and sends that withstatistics to analytics VM 104 (512), analytics VM 104 calculates byincoming signature and may distribute the calculation across otherAnalytics VM 104 in virtual network controller 100 (514), and analyticsVMs 104 compiles the result (516) and may optionally send the compiledresult to the web console 42 for display and/or further use (518). Theweb console may display the result (520).

FIG. 7 is a flowchart illustrating example operation of network devicesin accordance with the techniques of this disclosure. FIG. 7 illustratesone example operation of how latency may be calculated by a virtualnetwork controller, but the techniques of this disclosure are notlimited to this particular example. As shown in FIG. 7, VN switch agent172, on receiving one such packet, first calculates a hash (550) (suchas md5, sha1, etc.) of a part of packet that is path invariant anduniquely identifies the packet (such as, for example, the IP payload).This hash will be treated as the key, or signature, of the packet. Thiskey along with the switch identifier, which is unique to the VN switchagent 172, and which identifies which virtual switch 174 the packetpassed through, timestamp, ingress port, egress port, etc. as data willbe sent back to the analytics VM 104 for further processing (555).Analytics VM 104 also receives this data from VN agents 172 of otherservers 170 (not shown in FIG. 5) (560).

On the expiry of the end time (plus some jitter), analytics VMs 104 willstart processing each packet by the hash keys. Such initial processingmay include gathering data per key, and forming a list of values of thekey and assigning a job per key (packet hash) to one of the analyticsVMs 104 across virtual network controller 100 (565). That is, eachoriginal packet yielded a unique hash, which can be used as the key toidentify each of the packets and their information. Analytics VM 104 mayobtain, for each hash, a list of switch ids, timestamps, ingress portsand egress ports, etc.

For each hash, analytics VM 104 will then sort the associated list bytimestamp (570) and construct the topology map that the packet hastraversed (virtual network topology map) based on the list of switch idsand timestamps, and match the topology map up with the known physicaltopology of the network (575). The virtual network topology map includesa topology map of virtual switches based on the virtual switch ids. Asanalytics VMs 104 are linearly scaled, each gets a part of the job toprocess and determine the results. Near consistency of the timestamp isassumed to allow the clock drifts.

Next, the analytic engine 320 identifies the source and destination ofthis packet represented by the hash, and this hash can be broken down asn-distinct flows (580). Then, on each of the flows, analytics VM 104generates the path list (590), which consists of {switch-1, switch-2 . .. switch-r}, which are the specific physical switches that the packettraversed. Analytics VM 104 generates a hash on this switch list orpath-map (595), which is used as the key for the subsequent calculation.For each path-map hash, the near consistent time that the packet tookfrom its source to its destination can be determined. The expected erroris also calculated, which will be used to calculate the jitter orlatency per path.

With the path-map hash, all the flows can be combined (600) detected fora path-map and from there analytics VM 104 can compute the statisticalmeasure of the latency (605). By combining across the packet classifier,analytics VMs 104 can determine the real latency by evaluating minimum,maximum, mean and standard deviation per path in this network.

Using a collection of such latency information, virtual networkcontroller 100 can identify places in the physical network that are slowor where bottlenecks in traffic are occurring. Such a bottleneck may beindicative of a problem with the physical network, such as, for example,a deteriorated cable. Identifying such problems in the physical networkwithout having to run specific testing on each of the components of thenetwork saves time and money, and can help ensure that the networkperforms optimally and without interruption.

Additionally, the method can be used with any sets of physical switchesprovided that for each physical switch in the set there is an associatedVN-agent capable of receiving the PC, identifying (and hashing) theidentified packets, and forwarding it to an analytics engine for furtherprocessing as described above.

Various embodiments are described herein, including methods andtechniques. Techniques of this disclosure may also be used in an articleof manufacture that includes a non-transitory computer readable mediumon which computer-readable instructions for carrying out embodiments ofthe inventive technique are stored. The computer readable medium mayinclude, for example, semiconductor, magnetic, opto-magnetic, optical,or other forms of computer readable medium for storing computer readablecode. Further, the invention may also cover apparatuses for practicingembodiments of the invention. Such apparatus may include circuits,dedicated and/or programmable, to carry out operations pertaining toembodiments of the invention. Examples of such apparatus include ageneral purpose computer and/or a dedicated computing device whenappropriately programmed and may include a combination of acomputer/computing device and dedicated/programmable hardware circuits(such as electrical, mechanical, and/or optical circuits) adapted forthe various operations pertaining to embodiments of the invention.

FIG. 8 is a block diagram illustrating an example device 190 forcomputing latency in a physical network, in accordance with one or moreaspects of the present disclosure. FIG. 8 illustrates only oneparticular example of computing device 190, and many other examples ofcomputing device 190 may be used in other instances. Computing device190 may represent, for example, any of servers 12, TOR switches 16,chassis switches 18, virtual network controller 22, web console 42, orIFMAP server 26 of FIGS. 1-2, for example.

As shown in the specific example of FIG. 8, computing device 190includes one or more processors 200, one or more communication units202, one or more input devices 204, one or more output devices 206, andone or more storage devices 208. Computing device 190, in the specificexample of FIG. 8, further includes operating system 210, virtualizationmodule 212, and one or more applications 214A-214N (collectively“applications 214”). Each of components 200, 202, 204, 206, and 208 maybe interconnected (physically, communicatively, and/or operatively) forinter-component communications. As one example in FIG. 8, components200, 202, 204, 206, and 208 may be coupled by one or more communicationchannels 216. In some examples, communication channels 216 may include asystem bus, network connection, interprocess communication datastructure, or any other channel for communicating data. Virtualizationmodule 212 and applications 214, as well as operating system 210 mayalso communicate information with one another as well as with othercomponents in computing device 190. Virtualization may allow thefunctions of these components to be distributed over multiple machinesor multiple virtual machines, while a hypervisor gives the appearance ofsingle component.

Processors 200, in one example, are configured to implementfunctionality and/or process instructions for execution within computingdevice 190. For example, processors 200 may be capable of processinginstructions stored in storage devices 208. Examples of processors 200may include, any one or more of a microprocessor, a controller, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field-programmable gate array (FPGA), or equivalentdiscrete or integrated logic circuitry.

One or more storage devices 208 may be configured to store informationwithin computing device 190 during operation. Storage devices 208, insome examples, are described as a computer-readable storage medium. Insome examples, storage devices 208 are a temporary memory, meaning thata primary purpose of storage devices 208 is not long-term storage.Storage devices 208, in some examples, are described as a volatilememory, meaning that storage devices 208 do not maintain stored contentswhen the computer is turned off. Examples of volatile memories includerandom access memories (RAM), dynamic random access memories (DRAM),static random access memories (SRAM), and other forms of volatilememories known in the art. In some examples, storage devices 208 areused to store program instructions for execution by processors 200.Storage devices 208, in one example, are used by software orapplications running on computing device 190 (e.g., operating system210, virtualization module 212 and the like) to temporarily storeinformation during program execution.

Storage devices 208, in some examples, also include one or morecomputer-readable storage media. Storage devices 208 may be configuredto store larger amounts of information than volatile memory. Storagedevices 208 may further be configured for long-term storage ofinformation. In some examples, storage devices 208 include non-volatilestorage elements. Examples of such non-volatile storage elements includemagnetic hard discs, tape cartridges or cassettes, optical discs, floppydiscs, flash memories, or forms of electrically programmable memories(EPROM) or electrically erasable and programmable memories (EEPROM).

Computing device 190, in some examples, also includes one or morecommunication units 202. Computing device 190, in one example, utilizescommunication units 202 to communicate with external devices.Communication units 202 may communicate, in some examples, by sendingdata packets over one or more networks, such as one or more wirelessnetworks, via inbound and outbound links. Communication units 202 mayinclude one or more network interface cards (IFCs), such as an Ethernetcard, an optical transceiver, a radio frequency transceiver, or anyother type of device that can send and receive information. Otherexamples of such network interfaces may include Bluetooth, 3G and WiFiradio components. In some examples, computing device 190 utilizescommunication units 202 to communicate with other network devices, suchas to send or receive packet signatures as described herein.

Computing device 190, in one example, also includes one or more inputdevices 204. Input devices 204, in some examples, are configured toreceive input from a user through tactile, audio, or video feedback.Examples of input devices 204 include a presence-sensitive display, amouse, a keyboard, a voice responsive system, video camera, microphoneor any other type of device for detecting a command from a user. In someexamples, a presence-sensitive display includes a touch-sensitivescreen.

One or more output devices 206 may also be included in computing device190. Output devices 206, in some examples, are configured to provideoutput to a user using tactile, audio, or video stimuli. Output devices206, in one example, include a presence-sensitive display, a sound card,a video graphics adapter card, or any other type of device forconverting a signal into an appropriate form understandable to humans ormachines. Additional examples of output devices 206 include a speaker, acathode ray tube (CRT) monitor, a liquid crystal display (LCD), or anyother type of device that can generate intelligible output to a user.

Computing device 190 may include operating system 212. Operating system212, in some examples, controls the operation of components of computingdevice 190. For example, operating system 212, in one example,facilitates the communication of modules applications 214 withprocessors 200, communication units 202, input devices 204, outputdevices 206, and storage devices 210. Applications 214 may each includeprogram instructions and/or data that are executable by computing device190. As one example, application 214A may include instructions thatcause computing device 190 to perform one or more of the operations andactions described in the present disclosure.

In accordance with techniques of the present disclosure, computingdevice 190 may operate in accordance with the example processesdescribed in FIGS. 6-7.

The techniques described in this disclosure may be implemented, at leastin part, in hardware, software, firmware, or any combination thereof.For example, various aspects of the described techniques may beimplemented within one or more processors, including one or moremicroprocessors, digital signal processors (DSPs), application specificintegrated circuits (ASICs), field programmable gate arrays (FPGAs), orany other equivalent integrated or discrete logic circuitry, as well asany combinations of such components. The term “processor” or “processingcircuitry” may generally refer to any of the foregoing logic circuitry,alone or in combination with other logic circuitry, or any otherequivalent circuitry. A control unit including hardware may also performone or more of the techniques of this disclosure.

Such hardware, software, and firmware may be implemented within the samedevice or within separate devices to support the various techniquesdescribed in this disclosure. In addition, any of the described units,modules or components may be implemented together or separately asdiscrete but interoperable logic devices. Depiction of differentfeatures as modules or units is intended to highlight differentfunctional aspects and does not necessarily imply that such modules orunits must be realized by separate hardware, firmware, or softwarecomponents. Rather, functionality associated with one or more modules orunits may be performed by separate hardware, firmware, or softwarecomponents, or integrated within common or separate hardware, firmware,or software components.

The techniques described in this disclosure may also be embodied orencoded in an article of manufacture including a computer-readablestorage medium encoded with instructions. Instructions embedded orencoded in an article of manufacture including a computer-readablestorage medium encoded, may cause one or more programmable processors,or other processors, to implement one or more of the techniquesdescribed herein, such as when instructions included or encoded in thecomputer-readable storage medium are executed by the one or moreprocessors. Computer readable storage media may include random accessmemory (RAM), read only memory (ROM), programmable read only memory(PROM), erasable programmable read only memory (EPROM), electronicallyerasable programmable read only memory (EEPROM), flash memory, a harddisk, a compact disc ROM (CD-ROM), a floppy disk, a cassette, magneticmedia, optical media, or other computer readable storage media. In someexamples, an article of manufacture may include one or morecomputer-readable storage media.

A computer-readable storage medium comprises a non-transitory medium.The term “non-transitory” indicates that the storage medium is notembodied in a carrier wave or a propagated signal. In certain examples,a non-transitory storage medium may store data that can, over time,change (e.g., in RAM or cache).

Various examples have been described. These and other examples arewithin the scope of the following claims.

1. A method for determining latency of a physical network path in anetwork, the method comprising: receiving, by a virtual networkcontroller, a plurality of messages from a plurality of network devicesin a network, wherein each of the messages includes (1) a packetsignature comprising a hash of an invariant portion of an originalpacket that uniquely identifies the original packet, (2) an identifierof one of the plurality of network devices from which the respectivemessage was received, and (3) a timestamp indicating a time an originalpacket was processed by the network device from which the respectivemessage was received; identifying, by the virtual network controller,two or more of the plurality of messages having a common packetsignature; determining, by the virtual network controller, a latency ofa physical network path in the network based on analysis of contents ofthe identified messages having a common packet signature.
 2. The methodof claim 1, wherein the plurality of network devices comprise aplurality of virtual network agents executing on a plurality of serverdevices in the network, wherein the identifier of the network devicecomprises an identifier of a virtual switch associated with a virtualnetwork agent.
 3. The method of claim 2, wherein different ones of theplurality of server devices receive the same original packet atdifferent times.
 4. The method of claim 1, wherein the identifyingcomprises identifying, by a plurality of distributed analytics virtualmachines executing on the virtual network controller, the two or more ofthe plurality of messages having a common packet signature, and whereinthe determining comprises determining at least in part by the pluralityof distributed analytics virtual machines analyzing contents of theplurality of messages to determine the latency.
 5. The method of claim1, further comprising prior to receiving the plurality of messages,sending, by the virtual network controller, one or more commands to theplurality of network device indicating characteristics of packets to beanalyzed and a time range in which to collect packets matching thecharacteristics, wherein receiving the plurality of messages comprisesreceiving the plurality of messages responsive to sending the one ormore commands.
 6. The method of claim 1, further comprising: by thevirtual network controller: sorting contents of the plurality ofmessages by signature; obtaining for each signature, a list of networkdevice identifiers and timestamps; and sorting the list for eachsignature by timestamp.
 7. The method of claim 6, further comprising: bythe virtual network controller: for a given signature, constructing avirtual network topology map that the original packet traversed based onthe list of list of virtual network device identifiers and timestampsassociated with that signature; and matching the virtual networktopology map to a known physical topology of the network.
 8. The methodof claim 7, further comprising: identifying distinct flows associatedwith one of the signatures; generating a path map for one of theidentified flows comprising a list of physical network devices that apacket in the flows traversed based on the known physical topology ofthe network; generating a hash on the path map; and for each path maphash, determining a time that a packet took in traversing a physicalnetwork path from a source to a destination.
 9. The method of claim 8,further comprising: computing a statistical measure of the latency ofthe physical network path.
 10. A method comprising: receiving from avirtual network controller, by a network device, information specifyingpacket characteristics of packets to be analyzed; receiving a packet;responsive to determining that the packet matches the specifiedcharacteristics, and by a virtual network agent executing on the networkdevice, determining a hash of an invariant portion of the packet thatuniquely identifies the packet to obtain a packet signature; andforwarding, to the virtual network controller, a message that specifies:(1) the packet signature, (2) an identifier of the network device, and(3) a timestamp indicating a time the packet was processed by thenetwork device.
 11. The method of claim 10, wherein the identifier ofthe network device comprises an identifier of a virtual switchassociated with a virtual network agent.
 12. The method of claim 10,wherein the invariant portion of the packet comprises a payload of thepacket.
 13. The method of claim 10, wherein the received informationspecifies a time period during which to apply the packetcharacteristics.
 14. The method of claim 10, wherein the packetcharacteristics comprise a source address and destination address of thepackets to be monitored.
 15. A computer-readable storage mediumcomprising instructions for causing a programmable processor to: receivea plurality of messages from a plurality of network devices in anetwork, wherein each of the messages includes (1) a packet signaturecomprising a hash of an invariant portion of an original packet thatuniquely identifies the original packet, (2) an identifier of one of theplurality of network devices from which the respective message wasreceived, and (3) a timestamp indicating a time an original packet wasprocessed by the network device from which the respective message wasreceived; identify two or more of the plurality of messages having acommon packet signature; and determine a latency of a physical networkpath in the network based on analysis of contents of the identifiedmessages having a common packet signature.
 16. A virtual networkcontroller comprising: one or more processors; a plurality of virtualmachines executed by the processors to receive a plurality of messagesfrom a plurality of network devices in a network, wherein each of themessages includes (1) a packet signature comprising a hash of aninvariant portion of an original packet that uniquely identifies theoriginal packet, (2) an identifier of one of the plurality of networkdevices from which the respective message was received, and (3) atimestamp indicating a time an original packet was processed by thenetwork device from which the respective message was received; and aplurality of analytics virtual machines, wherein the plurality ofvirtual machines identify two or more of the plurality of messageshaving a common packet signature, and determine a latency of a physicalnetwork path in the network based on analysis of contents of theidentified messages having a common packet signature.
 17. The virtualnetwork controller of claim 16, wherein the plurality of analyticsvirtual machines sort contents of the plurality of messages bysignature, obtain for each signature, a list of network deviceidentifiers and timestamps, and sort the list for each signature bytimestamp.
 18. The virtual network controller of claim 17, wherein theplurality of analytics virtual machines, for a given signature,construct a virtual network topology map that the original packettraversed based on the list of list of virtual network deviceidentifiers and timestamps associated with that signature, and match thevirtual network topology map to a known physical topology of thenetwork.
 19. The virtual network controller of claim 18, wherein theplurality of analytics virtual machines identify distinct flowsassociated with one of the signatures, generate a path map for one ofthe identified flows comprising a list of physical network devices thata packet in the flows traversed based on the known physical topology ofthe network, generate a hash on the path map, and for each path maphash, determine a time that a packet took in traversing a physicalnetwork path from a source to a destination.
 20. The virtual networkcontroller of claim 16, wherein the plurality of analytics virtualmachines compute a statistical measure of the latency of the physicalnetwork path.
 21. A system comprising: a virtual network controllercomprising: one or more processors; a plurality of virtual machinesexecuted by the processors; and a plurality of network devicescomprising one or more processors, wherein the plurality of networkdevices receive from the virtual network controller, informationspecifying packet characteristics of packets to be analyzed receiving apacket, wherein the plurality of virtual machines receive a plurality ofmessages from the plurality of network devices, wherein each of themessages includes (1) a packet signature comprising a hash of aninvariant portion of an original packet that uniquely identifies theoriginal packet, (2) an identifier of one of the plurality of networkdevices from which the respective message was received, and (3) atimestamp indicating a time an original packet was processed by thenetwork device from which the respective message was received; whereinthe virtual network controller further comprises a plurality ofanalytics virtual machines that identify two or more of the plurality ofmessages having a common packet signature, and determine a latency of aphysical network path in the network based on analysis of contents ofthe identified messages having a common packet signature; wherein theplurality of network devices comprise a virtual network agent executingon the processors that, responsive to determining that the packetmatches the specified characteristics, determining a hash of aninvariant portion of the packet that uniquely identifies the packet toobtain a packet signature, and forward, to the virtual networkcontroller, a message that specifies: (1) the packet signature, (2) anidentifier of the network device, and (3) a timestamp indicating a timethe packet was processed by the network device.
 22. The system of claim21, wherein plurality of network devices comprise a plurality of serverdevices each comprising a respective virtual switch and a respectivevirtual network agent, and wherein the identifier specified by themessage comprises an identifier of a virtual switch associated with avirtual network agent.